Systems and Methods for Matching Visual Object Components

ABSTRACT

Systems and methods for modeling the occurrence of common image components (e.g., sub-regions) in order to improve visual object recognition are disclosed. In one example, a query image may be matched to a training image of an object. A matched region within the training image to which the query image matches may be determined and a determination may be made whether the matched region is located within an annotated image component of the training image. When the matched region matches only to the image component, an annotation associated with the component may be identified. In another example, sub-regions within a plurality of training image corpora may be annotated as common image components including associated information (e.g., metadata). Matching sub-regions appearing in many training images of objects may be down-weighted in the matching process to reduce possible false matches to query images including common image components.

FIELD

This disclosure relates to object recognition, and in examples, torecognition of object components.

BACKGROUND

The ability to recognize objects can be used in many applications. Whilehumans may easily identify a given object within an image or video, thesame task may be more complex for computer vision systems. Numerousapproaches and techniques have been developed to attempt to recognizeobjects using computer vision systems. One example includes training arecognition engine, or matcher, using a set of training, or reference,images.

A matcher may have knowledge of a set of training images used forrecognizing objects. For example, training corpora may contain images ofa set of objects, with one or more images of visual appearances of theobject, as well as metadata identifying the objects. Given an image ofan object to be recognized, the matcher may retrieve training imagessimilar to the object and determine a match. Often times correspondingmetadata of the matched training images may be returned, such as a type,name, title, etc. of the recognized object. Additionally, a matchedregion of the training image, to which the image of the object matches,may also be identified.

SUMMARY

This disclosure may disclose, inter alia, systems and methods forlearning and matching visual object components.

In one example, a method for identifying an annotated image componentmatching a query image is provided. The method includes, but is notlimited to, receiving a query image. The method further includesmatching the query image to an object using a visual object recognitionmodule. The visual object recognition module may be configured toutilize information from a plurality of training image corpora to matchthe query image to a training image of the object, for example. Themethod also includes determining a matched region within the trainingimage to which the query image matches using the visual objectrecognition module. Additionally, the method includes determiningwhether the matched region is located within an annotated imagecomponent of the training image. In one example, the annotated imagecomponent may be a sub-region of the training image including anassociated annotation describing the sub-region. The method furtherincludes identifying an annotation associated with the annotated imagecomponent when the matched region is located within the annotated imagecomponent of the training image.

In another example, a non-transitory computer-readable medium withinstructions stored thereon is provided. The instructions may beexecutable by a computing device. The instructions may be executable forreceiving a query image and matching the query image to an object usinga visual object recognition module. The visual object recognition modulemay be configured to utilize information from a plurality of trainingimage corpora, for example, to match the query image to a training imageof the object. The instructions may be further executable fordetermining a matched region within the training image to which thequery image matches using the visual object recognition module. Theinstructions may also be executable for determining whether the matchedregion is located within an annotated image component of the trainingimage. The annotated image component may be a sub-region of the trainingimage, for example, and may include an associated annotation describingthe sub-region. According to the executable instructions, an annotationassociated with the annotated image component may be identified when thematched region is located within an annotated image component of thetraining image.

In another example, a system is provided. The system involves a memoryand a processor coupled to the memory. The system further includesinstructions, executable by the processor, stored in the memory. Theinstructions may be executable for receiving a query image and matchingthe query image to an object using a visual object recognition module.The visual object recognition module may be configured to utilizeinformation from a plurality of training image corpora, for example, tomatch the query image to a training image of the object. Theinstructions may be further executable for determining a matched regionwithin the training image to which the query image matches using thevisual object recognition module. The instructions may also beexecutable for determining whether the matched region is located withinan annotated image component of the training image. The annotated imagecomponent may be a sub-region of the training image, for example, andmay include an associated annotation describing the sub-region.According to the executable instructions, an annotation associated withthe annotated image component may be identified when the matched regionis located within an annotated image component of the training image.

The foregoing summary is illustrative only and is not intended to be inany way limiting. In addition to the illustrative aspects, embodiments,and features described above, further aspects, embodiments, and featureswill become apparent by reference to the figures and the followingdetailed description.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 illustrates an example system.

FIG. 2 is an example block diagram of a method to match a query image toan annotated image component, in accordance with at least someembodiments described herein.

FIG. 3A illustrates an example of a query image and a matched trainingimage.

FIG. 3B illustrates another example of a query image and a matchedtraining image.

FIG. 4 is an example block diagram of a method to discover and annotateobject components, in accordance with at least some embodimentsdescribed herein.

FIG. 5A is an example directed acyclic graph of possible objectcomponent relationships.

FIG. 5B illustrates an example of annotating an object component.

FIG. 6 is an example block diagram of a method to assign a weightingfactor to common image components, in accordance with at least someembodiments described herein.

FIG. 7A illustrates an example of common image components.

FIG. 7B illustrates another example of common image components.

FIG. 8 is a functional block diagram illustrating an example computingdevice used in a computing system that is arranged in accordance with atleast some embodiments described herein.

FIG. 9 is a schematic illustrating a conceptual partial view of anexample computer program product that includes a computer program forexecuting a computer process on a computing device, arranged accordingto at least some embodiments presented herein.

DETAILED DESCRIPTION

In the following detailed description, reference is made to theaccompanying figures, which form a part hereof. In the figures, similarsymbols typically identify similar components, unless context dictatesotherwise. The illustrative embodiments described in the detaileddescription, figures, and claims are not meant to be limiting. Otherembodiments may be utilized, and other changes may be made, withoutdeparting from the scope of the subject matter presented herein. It willbe readily understood that the aspects of the present disclosure, asgenerally described herein, and illustrated in the figures, can bearranged, substituted, combined, separated, and designed in a widevariety of different configurations, all of which are explicitlycontemplated herein.

This disclosure may disclose, inter alia, systems and methods forautomatically learning and matching visual object components. Thesystems and methods may be directed to modeling the occurrence of commonimage components (e.g., sub-regions) in order to improve visual objectrecognition.

In one example, a query image may be received and matched to an objectusing a visual object recognition module. The visual object recognitionmodule may be configured to utilize information from a plurality oftraining image corpora, for example, to match the query image to one ormore training images of the object. A matched region within the trainingimage to which the query image matches may also be determined by thevisual object recognition module. Moreover, a determination may be madewhether the matched region is located within an annotated imagecomponent of the training image. As an example, an annotated imagecomponent may be a sub-region of a training image including anannotation describing the sub-region.

In one example, when the matched region matches only to the imagecomponent (or a portion of an object of the image) rather than the wholeobject of the image, an annotation associated with the component may beidentified and returned to a user. For example, when the matched regionis located within the annotated image component of the training image,an annotation associated with the annotated image component may beidentified. In addition, when a percentage of the matched region locatedwithin the annotated image component is less than a percentagethreshold, an annotation associated with an entirety of the trainingimage may be identified.

In another example, a plurality of training image corpora may bereceived. Cross-corpus image region matching may be performed among theplurality of training image corpora to discover common image components.Matched regions within the plurality of training image corpora may beannotated as sub-regions within the training images including associatedinformation (e.g., metadata).

In yet another example, weighting factors influencing a matching processof the visual object recognition module may be adjusted. For example,common image components that frequently appear in many training imagesof objects may be down-weighted in the matching process to reducepossible false matches to query images including common imagecomponents.

Referring now to the figures, FIG. 1 illustrates an example system 100.One or more parts of the system 100 may be used to perform methods forlearning and matching visual object components. The system 100 mayinclude a processor 102 coupled to a memory 104. Additionally the system100 may include a plurality of training image corpora 106, a query image108, and a visual object recognition module 110, all of which may becoupled to the processor 102 and the memory 104.

The processor 102 may be any type of processor, such as amicroprocessor, digital signal processor (DSP), multicore processor,etc., coupled to the memory 104. The memory 104 may be any type ofmemory, such as volatile memory like random access memory (RAM), dynamicrandom access memory (DRAM), static random access memory (SRAM), ornon-volatile memory like read-only memory (ROM), flash memory, magneticor optical disks, or compact-disc read-only memory (CD-ROM), among otherdevices used to store data or programs on a temporary or permanentbasis. In one example, the memory 104 may include non-transitorycomputer-readable media.

In one embodiment, the plurality of training image corpora 106 mayinclude or describe a set of objects. For each object, there may be oneor more training images containing visual appearances of the object, aswell as metadata associated with the object (e.g., type, name, etc.).

In some examples, information within the plurality of training imagecorpora 106 may be processed for use by the visual object recognitionmodule 110. For example, raw image data may be computed for comparingimage similarity within a matching routine. Alternatively, matching maybe performed based on image templates as an alternative to (or inaddition to) matching raw image data. Matching image templates may be amore effective method than matching raw image data in terms ofdetermining image similarity, for example.

In one example, for each image (e.g., training images of the pluralityof training image corpora 106 and the query image 108), descriptors forimage interest points may be extracted and an image template may bebuilt. A descriptor may include information extracted from local pixelsaround an image interest point. An image interest point may be a pointin the image that can have a clear definition (e.g., may bemathematically well-founded), can have a well-defined position in imagespace, can have a local image structure that is rich in terms of localinformation contents, and is stable under local and global perturbationsin the image domain. For example, image interest points may be corners,text within an image, blobs, etc., and may be located using opticalcharacter recognition (OCR), edge detection techniques, analyzingpredetermined positions, among other possibilities. In some examples, atemplate may include the extracted information of the image and a set ofdescriptors of all interest points in the image.

The query image 108 may be or include one or more images submitted byone or more users. The query image 108 may be of any size, shape, and/orform. Similarly, the query image 108 may be two-dimensional (e.g., aphotograph, a figures, a picture) or three-dimensional (e.g. ahologram). The query image 108 may be captured by optical devices(cameras, mirrors, lenses, etc.) or otherwise input by a system or user(e.g., visual sketch drawn or otherwise rendered on an input surface).In one example, the query image 108 may be a frame or multiple framescaptured from a moving image or video.

In some examples, the query image 108 may include an image provided inorder to perform content-based image retrieval. Content from within thequery image 108 may be analyzed and one or more training images matchingthe content of the query image 108 may be returned. For example, contentof the query image 108 may refer to colors, shapes, textures, or otherinformation derived from an image. In some examples, the query image 108may include an image submitted by a user containing metadata such askeywords, tags, or other descriptions associated with the image.

The visual object recognition module 110 may include or be configured tooperate according to one or more programming instructions, for example,computer executable and/or logic implemented instructions to performfunctions or steps. Additionally, the visual object recognition module110 may be circuitry wired to perform one or more programminginstructions.

In one example, the visual object recognition module 110 may have accessto and interact with a set of training images of the plurality oftraining image corpora 106. Given a query image 108, the visual objectrecognition module 110 may retrieve and output training images that aresimilar to the query image 108. Additionally, for each similar trainingimage, a match score may be provided. For example, the match score maybe computed based on a combination of one or more of a number of matcheddescriptors between the query image 108 and a training image, a computedsimilarity between raw image data of the query image 108 and a trainingimage, or other potential image comparison algorithms or methods.Moreover, the visual object recognition module 110 may outputcorresponding metadata of the matched training images.

FIG. 2 is an example block diagram of a method 200 to match a queryimage to an annotated image component, in accordance with at least someembodiments described herein. The method 200 shown in FIG. 2 presents anembodiment of a method that may, for example, be used by the system 100of FIG. 1. Method 200 may include one or more operations, functions, oractions as illustrated by one or more of blocks 201-213. Although theblocks are illustrated in a sequential order, these blocks may also beperformed in parallel, and/or in a different order than those describedherein. Also, the various blocks may be combined into fewer blocks,divided into additional blocks, and/or removed from the method, basedupon the desired implementation of the method.

In addition, for the method 200 and other processes and methodsdisclosed herein, the flowchart shows functionality and operation of onepossible implementation of present embodiments. In this regard, eachblock may represent a module, a segment, or a portion of program code,which includes one or more instructions executable by a processor forimplementing specific logical functions or steps in the process. Theprogram code may be stored on any type of computer readable medium, forexample, such as a storage device including a disk or hard drive. Thecomputer readable medium may include non-transitory computer readablemedium, for example, such as computer-readable media that stores datafor short periods of time like register memory, processor cache andrandom access memory (RAM). The computer readable medium may alsoinclude non-transitory media, such as secondary or persistent long termstorage, like read only memory (ROM), optical or magnetic disks,compact-disc read only memory (CD-ROM), for example. The computerreadable media may also be any other volatile or non-volatile storagesystems. The computer readable medium may be considered a computerreadable storage medium, for example, or a tangible storage device.

In addition, for the method 200 and other processes and methodsdisclosed herein, each block in FIG. 2 may represent circuitry that iswired to perform the specific logical functions in the process.

Initially, at block 201, the method 200 includes receiving a queryimage. The query image may be an image collected and stored by a systemwhich receives query images from users. In one example, the query imagemay be a user submitted image for which information is desired. A usermay submit a query image in order to determine images matching the queryimage, or to determine information about the query image, for example.Images matching the query image may include metadata associated withand/or describing the matched images that may be output to the user.Additionally, the query image may be a query comprising multiple images.

The query image may be of one or more image file formats. For example,the query image may be an image file composed of either pixel or vectordata that is rasterized to pixels when displayed in a vector graphicdisplay. Example image file formats include JPEG, JPEG 2000, TIFF, RAW,PNG, GIF, BMP, among other possible image file formats. Similarly, thequery image may optionally be compressed using a variety of imagecompression techniques. In some examples, the method 200 includesdecompressing the query image for use in conjunction with the method 200and/or other systems and methods disclosed.

At block 203, the method 200 includes matching the query image to anobject using a visual object recognition module. Given a query image,the visual object recognition module may retrieve and output one or moretraining images matching the query image. The visual object recognitionmodule may utilize information from a plurality of training imagecorpora. For example, the plurality of training image corpora mayinclude one or more training images associated with a plurality ofobjects.

In one example, a query image may be an image of an object differingfrom training images of the object within the training corpus. The queryimage may, for example, be matched to a training image from theplurality of objects of the training corpus with a variation inillumination of the object. The query image may have been captured underdifferent lighting settings compared with the illumination of the objectwithin the training images of the object. Similarly, in another example,the query image may be an image of an object with a variation in pose orshape of the object as compared to the pose or shape of the object in agiven training image.

In another example, the visual object recognition module may match aregion or multiple regions of a query image to one or more trainingimages. For example, a query image may be processed to remove backgroundinformation from the query image or split into multiple regions ofseparate objects. In one example, a number of similar image descriptorsbetween a training image and the query image may be above a threshold.Accordingly, the training image may be determined to be a match to thequery image. Other example methods and techniques used for imagematching include template matching, normalized cross-correlation, ormulti-variate cross-correlation.

In yet another example, match scores may be associated with matchesbetween the query image and one or more training images (or one or morematched regions of the one or more training images) of the plurality ofobjects. The match score may indicate a level of similarity between aquery image and a matching training image (or matched region of atraining image), for example. Matches may be prioritized or ranked basedon the match scores. In one example, a training image associated with amaximum match score may be selected as a match the query image. Inanother example, the similarity may be computed based on a number ofmatched descriptors between a query image and a training image. However,other methods of determining similarity are also contemplated. In oneexample, an image distance measure may compare the similarity of twoimages in various dimensions such as color, texture, shape, and others.For example, color histograms of images may be compared, where a colorhistogram is computed by quantizing the colors within an image andcounting the number of pixels of each color. In another example, anintensity-based method may be used to compare patterns in a query imageand a training image using correlation metrics.

At block 205, the method 200 includes determining a matched regionwithin a training image to which the query image matches using thevisual object recognition module. For example, the visual objectrecognition module may also output a matched region and/or matcheddescriptors associated with a query image and one or more matchingtraining images. In one example, the matched region may be determined byforming a boundary (e.g., a rectangle, circle, or free-form shape)around matched descriptors with a level of similarity or quality ofmatch above a threshold. The various methods described above withreference to determining similarity between images (e.g., image distancemeasure, color methods, intensity-based methods, etc.) may also beapplied at the image descriptor level. In another example, the queryimage may only match a portion of the training image. Similarly, amatched region within a training image may be determined by forming aboundary around matched descriptors. In a further example, multiplematched regions may be determined.

At block 207, the method 200 includes determining whether the matchedregion is located within an annotated image component of the trainingimage. The annotated image component may be, for example, a sub-regionof the training image including an annotation describing the sub-region.Additionally, a training image may include multiple annotated imagecomponents, and a determination may be made whether the matched regionis located within one or more annotated image components.

In one example, a tolerance may be used for determining whether thematched region is located within an annotated image component. Thematched region may be determined to be located within an annotated imagecomponent if the matched region lies largely within the annotated imagecomponent. For example, the determination may be made based on apercentage of the matched region located within an annotated imagecomponent. Similarly, the determination may be made based on a number ofmatched descriptors of the matched region located within an annotatedimage component.

At block 209, a decision may be made based on the determination at block207. If the matched region is determined to be located within anannotated image component of the training image, block 211 of the method200 may be executed. Alternatively, if the matched region is determinedto not be located within an annotated image component, block 213 of themethod 200 may be executed. For example, the matched region may match toan entirety of the training image or multiple regions of the trainingimage.

At block 211, the method 200 includes identifying an annotationassociated with the annotated image component. Accordingly, theannotation may be returned responsive to the query image being received.The query image may be determined to be a match to the image component,and metadata of the component may be identified and output instead ofmetadata of the object which the matching training image depicts.

In one example, the annotation associated with the annotated imagecomponent may be identified when a percentage of the matched regionlocated within the annotated image component is greater than apercentage threshold. For example, the percentage threshold may be 50%,75%, 90%, or above 75%. Likewise, the annotation associated with theannotated image component may be identified when a number of descriptorsof the matched region within the annotated image component is above anumber threshold.

At block 213, the method 200 includes identifying an annotationassociated with an entirety of the training image. Accordingly, theannotation may be returned responsive to the query image being received.The query image may be determined to be a match to the training image asa whole, and metadata of the object which the training image depicts maybe identified and output.

In one example, the annotation associated with the entirety of thetraining image may be identified when a percentage of the matched regionlocated within the annotated image component is less than a percentagethreshold. Likewise, the annotation associated with entirety of thetraining image may be identified when a number of descriptors of thematched region within the annotated image component is above a numberthreshold.

According to an example of the method 200, the query image may bereceived from a client device. The identified annotation associated withthe image component of the training image may be sent to the clientdevice when a determination may be made that, for example, the matchedregion is located within an annotated image component of the trainingimage.

FIG. 3A illustrates an example of a query image 301 a and a matchedtraining image 303. The query image 301 a may be matched to the trainingimage 303, for example, by a visual recognition module. The visualrecognition module may also determine a matched region 305 a within thetraining image 303 to which the query image 301 a (or a region of thequery image 301 a) matches.

In one example, a determination may be made whether the matched region305 a is located within an annotated image component 307 of the trainingimage 303. The annotated image component 307 may be a sub-region of thetraining image 303 and may include an associated component annotation309 describing the sub-region. For example, the annotated imagecomponent 307 may be an image of a can of soda on the cover of a bookabout soda. The component annotation 309 of the annotated imagecomponent 307 may be metadata describing a logo on the can of soda.

Although the matched region 305 a in FIG. 3A may include the annotatedimage component 307, the matched region 305 a, as illustrated, does notlie within (or substantially within) the annotated image component 307.Accordingly, a system, such as the system 100 of FIG. 1, responsive toreceiving the query image 301 a, may return an object annotation 311associated with an object (e.g., a book titled “A book about soda”)described by the entirety of the training image 303. In this example,the object annotation 311 may be identified rather than the componentannotation 309.

FIG. 3B illustrates another example of a query image 301 b and a matchedtraining image 303. The query image 301 b may be matched to the trainingimage 303, for example, by a visual object recognition module. Thevisual object recognition module may also determine a matched region 305b within the training image 303 to which the query image 301 b (or aregion of the query image 301 b) matches.

In one example, a determination may be made whether the matched region305 b is located within annotated image component 307 of the trainingimage 303. The matched region 305 b in FIG. 3B may lie within (orsubstantially within) the annotated image component 307. Accordingly, asystem, such as the system 100 of FIG. 1, responsive to receiving thequery image 301 b, may return the component annotation 309 associatedwith the annotated object component 307. The component annotation 309may be returned rather than the object annotation 311. Thus, the queryimage 301 b may be matched to an object component (or informationassociated with the object component, e.g., soda), rather thanincorrectly matched to an object (or information associated with theobject, e.g., book) with a sub-region depicting the query image 301 b.

FIG. 4 is an example block diagram of a method 400 to discover andannotate object components, in accordance with at least some embodimentsdescribed herein. The method 400 shown in FIG. 4 presents an embodimentof a method that may, for example, be used by the system 100 of FIG. 1.Method 400 may include one or more operations, functions, or actions asillustrated by one or more of blocks 401-405. Although the blocks areillustrated in a sequential order, these blocks may also be performed inparallel, and/or in a different order than those described herein. Also,the various blocks may be combined into fewer blocks, divided intoadditional blocks, and/or removed from the method, based upon thedesired implementation of the method. Each block may represent a module,a segment, or a portion of program code, which includes one or moreinstructions executable by a processor for implementing specific logicalfunctions or steps in the process. In addition, each block in FIG. 4 mayrepresent circuitry that is wired to perform the specific logicalfunctions in the process.

Initially, at block 401, the method 400 includes receiving a pluralityof training image corpora. The plurality of training image corpora maydescribe a set of objects. For each object, there may be one or moretraining images containing visual appearances of the object, as well asmetadata associated with the object (e.g., type, name, etc.). Theplurality of training image corpora may be utilized by the visual objectrecognition module to match images. For example, the visual objectrecognition module may be trained based on images within the pluralityof training image corpora.

At block 403, the method 400 includes performing cross-corpus imageregion matching among the plurality of training image corpora. Commonobject components appearing in more than one image of the plurality oftraining image corpora may be discovered and identified. Common objectcomponents may include, for example, logos, famous artwork, letters in aspecific font, etc., appearing on different objects represented withinthe plurality of training image corpora. The plurality of training imagecorpora may be compared to determine similarities, and object componentswith a level of similarity above a threshold may be determined to becommon object components.

In one example, a matcher or visual object recognition module may bebuilt from all the training images of the plurality of training imagecorpora. Each training image may be matched using the matcher toidentify common sub-structure that appears in more than one trainingimage. For example, matched regions within matching training images ofthe plurality of training image corpora may be determined based on asimilarity between image descriptors of the matching training images. Inanother example, optical character recognition (OCR) may be used todetect text within training images. Recognized characters associatedwith training images may be compared and common image components may bedetermined based on a comparison of recognized characters betweentraining images.

At block 405, the method 400 includes, for matched regions within theplurality of training image corpora, annotating the matched regions assub-regions within the training images comprising associatedinformation. For example, the matcher may identify a matched region in afirst training image matching a second training image. An annotationassociated with the second training image may be associated with thematched region of the first training image. Similarly, the matcher mayidentify a matched region in a first training image matching asub-region of second training image for which a component annotation isassociated. The component annotation of the second training image may beassociated with the matched region of the first training image.

FIG. 5A is an example directed acyclic graph 500 of possible objectcomponent relationships. In FIG. 5A, a collection of vertices anddirected edges are illustrated. However, the vertices and directed edgesare not meant to be limiting, and are provided as examples modeling astructure. Structures associated with the systems and methods describedherein may depart from the example illustrated in FIG. 5A, and mayinclude more or less vertices and/or directed edges than illustrated.

FIG. 5A includes vertices or nodes representing training corpora 501a-d. The training corpora 501 a-d may comprise training images of a logo(501 a), a piece of artwork (501 b), a landmark (501 c), and a book (501d) respectively.

Additionally, the training corpora 501 a-d may be connected by directededges 503 a-e. A directed edge 503 a-e from a first training corpus to asecond corpus may represent the presence of an image (or portion of animage) from the first training corpus within an image from the secondcorpus. For example, directed edge 503 a may present the presence of animage from training corpus 501 a (e.g., a logo) within an image fromtraining corpus 501 c (e.g., a landmark). Similarly, an image of a bookcover from training corpus 501 d may include a logo, a piece of artwork,or a landmark image.

In one example, a visual object recognition module may match trainingimages of first training corpus against training images of a secondtraining corpus for each directed edge 503 a-e. If a match is found (asin the example of FIG. 5B), the matching region of a training image fromthe second training corpus may be annotated with the metadata of thetraining image from the first training corpus.

FIG. 5B illustrates an example of annotating an object component.Cross-corpus image region matching may be performed among the trainingcorpora 501 a-d. In one example, training images (or sub-regions oftraining images) from a first training corpus may be matched to trainingimages of a second training corpus. For example, a first training image507 d from training corpus 501 d may match to a second training image507 a from training corpus 501 a. The first training image 507 d and thesecond training image 507 a may each include associated information. Thefirst training image 507 d may include associated metadata 509 d, whilethe second training image 507 a may include associated metadata 509 a.

A matched region 511 within the first training image 507 d may bedetermined for the match. As an example, the first training image 507 dmay be an image of a book cover and the matched region 511 may be acomponent of the book cover including an image of a logo. In oneexample, the matched region 511 may be annotated as a sub-region withinthe first training image 507 d comprising associated information. Forexample, the matched region 511 may be annotated with metadata 509 a.

In another example, multiple matched regions with the first trainingimage 507 d may be determined for the match. The multiple matchedregions may each be annotated as sub-regions within the first trainingimage 507 d comprising associated information.

FIG. 6 is an example block diagram of a method 600 to assign a weightingfactor to common image components, in accordance with at least someembodiments described herein. The method 600 shown in FIG. 6 presents anembodiment of a method that may, for example, be used by the system 100of FIG. 1. Method 600 may include one or more operations, functions, oractions as illustrated by one or more of blocks 601-607. Although theblocks are illustrated in a sequential order, these blocks may also beperformed in parallel, and/or in a different order than those describedherein. Also, the various blocks may be combined into fewer blocks,divided into additional blocks, and/or removed from the method, basedupon the desired implementation of the method. Each block may representa module, a segment, or a portion of program code, which includes one ormore instructions executable by a processor for implementing specificlogical functions or steps in the process. In addition, each block inFIG. 6 may represent circuitry that is wired to perform the specificlogical functions in the process.

Initially, at block 601, the method 600 includes performing cross-corpusmatching among the plurality of training image corpora to identifycommon image components appearing in more than one training image.Common object components appearing in more than one training image ofthe plurality of training image corpora may be discovered andidentified.

Common object components may include, for example, logos, famousartwork, letters in a specific font, etc., appearing on differentobjects represented within the plurality of training image corpora. Textwithin training images may be a common image component. Optionally, thetext within multiple training images may be in similar fonts. Othersources of common object components appearing in multiple trainingimages include: stock images reused in images of different objects;common patterns such as grids, stars, circles, etc.; and common partssuch as a similar frame used in images of two different pieces ofartwork.

In one example, a matcher or visual object recognition module may bebuilt from all the training images of the plurality of training imagecorpora. Each training image may be matched using the matcher toidentify common sub-structure that appears in more than one trainingimage.

In another example, optical character recognition (OCR) may be used todetect text within training images. Recognized characters associatedwith training images may be compared and common image components may bedetermined based on a comparison of recognized characters betweentraining images.

In a further example, a matcher or visual object recognition module maybe built from a database of stock images. The matcher may be used todetect reused image sub-structure in the training images as common imagecomponents.

At block 603, the method 600 includes identifying image descriptorsappearing in more than one training image. Similar image descriptorswithin common image components may be determined. For example, an imagedescriptor within a common image component appearing in more than onetraining image may be annotated with additional information indicatingthat the image descriptor may be part of a common image component. Inanother example, an image descriptor within a common image component maybe annotated with additional information indicating the number oftraining images in which the image descriptor appears.

In a further example, image descriptors may be determined by examiningan image, identifying image interest points (e.g., corners, text, edges,etc.) that include distinctive regions in the image, and assigning anidentifier to the image interest point. In one instance, an imagedescriptor includes a numerical representation of pixels that fallwithin an image interest point (or image sub-region) of the image. Inone example, image descriptors include descriptions of visual featuresof contents in the image, such as characteristics including a shape, acolor, or a texture among others. The image descriptors may becompressed, and compressed image descriptors may be compared to identifysimilar image descriptors.

For example, a technique for compressing image descriptors includesproduct quantization. Image descriptors may comprise a number offloating point numbers (e.g., referred to as the dimensionality of thedescriptor). Product quantization may be performed to divide thedimensions into groups (e.g., such as evenly divided groups) andquantize or represent each group of dimensions using one of K exemplars.An exemplar may be chosen to minimize a sum of squared distances betweenthe descriptor dimension and a nearest neighbor exemplar. Exemplars maybe learned from unsupervised clustering algorithms, such as k-means, forexample. The k-means clustering algorithm may include a method ofcluster analysis that aims to partition n observations into k clustersin which each observation belongs to the cluster with the nearest mean.The k-means clustering algorithm may attempt to find a center of naturalclusters in the data.

In one example, in which a descriptor has 100 dimensions (and thus maycomprise 100 floating point values) product quantization may includedividing the 100 dimensions into groups, and each group may include fourto five dimensions. Thus, the 100 dimensions may be divided into 20groups with 5 dimensions per group (e.g., group 1 includes dimensions#1-5, group 2 includes dimensions #6-10, and so on). Productquantization represents each group separately using exemplars.

In one example, a list of exemplars may be provided within a codebook. Aclosest exemplar to the group of dimensions is identified, and selectedto represent the group. In the example above, group 1 includingdimensions #1-5 (e.g., five floating point values) may be represented bya single exemplar. Thus, instead of representing or storing fivefloating point values, a single exemplar may be used to represent andstore the group of dimensions. Using this method, the descriptorincluding 100 dimensions, which were divided into 20 groups can berepresented by a data representation using 20 exemplars. Thus, thedescriptor may be compressed from a data representation of 100 datapoint to 20 data points. The 20 data points may be compared across imagedescriptors of two or more potentially matching training images. In oneexample, if an image descriptor from a first training image matches animage descriptor from a second training image, the image descriptor maybe annotated with an additional data point indicating the imagedescriptor may match to another training image. Similarly, theadditional data point may be increased if the image descriptor may befound to be similar to image descriptors of more training images.

At block 605, the method 600 includes assigning a weighting factor toimage descriptors associated with the common image components. Insteadof treating all descriptors in an image equally, a weighting factor maybe assigned to each of the image descriptors within the image. Theweighting factor may affect a match score or measure of similaritybetween one or more image descriptors of training images. In oneexample, lowering the weighting factor of image descriptors within acommon image component may have the effect of lowering match scoreswhich (fully or partially) come from the common image component. Thus,matches may be suppressed in varying degrees depending on the portion ofa match that comes from common image components.

In one example, the weighting factor associated with all imagedescriptors of an image may be 1. Additionally, the weighting factorassociated with descriptors determined to be associated with a commonimage component may be assigned a weighting factor of alpha, where alphais less than 1. The value of the weighting factor alpha may be a fixedor predetermined value. In other examples, the value of the weightingfactor alpha may be a variable, dependent on one or more factors of asystem or method.

In one example, the value of the weighting factor alpha may bedetermined experimentally. A plurality of matching experiments may beperformed on a validation set using a series of weighting factor valuesfor alpha. The validation set may be a set of query images including oneor more common image components within the plurality of training imagecorpora, for example. Matching accuracies associated with results of theplurality of matching experiments may be determined. A matching accuracymay be determined for each value of alpha from the series of weightfactor values of alpha. The matching accuracy may be determined byanalyzing the results received for the set of query images. A value ofalpha corresponding to the weighting factor yielding the maximummatching accuracy may be selected and used for image descriptors ofcommon image components.

In another example, weighting factors for image descriptors of atraining image within a plurality of training image corpora may bedetermined using a term frequency—inverse document frequency (TF-IDF)formula. For example, the weighting factors for image descriptors of atraining image may be determined using the following formula:

(1/x)*log(D/d),

wherein x is a total number of image descriptors in the training image,D is a total number of training images in the plurality of trainingimage corpora, and d is a total number of training images in which theimage descriptor appears.

At block 607, the method 600 includes determining adjusted match scoresbased on the weighting factor. Multiple approaches exist on how theweight of descriptors may be applied to obtain an adjust match score.One possible example includes determining an adjust match score given anoriginal match score based on the following formula:

${S^{\prime} = {S*\frac{w_{matched}}{w_{image}}}},$

where S′ is an adjusted match score, S is a match score computed withoutassigning weighting factors to image descriptors, dw_(matched) is a sumof weighting factors of matched image descriptors within a trainingimage, and dw_(image) is a sum of weighting factors of image descriptorswithin the training image. Thus, the match score may be adjustedaccording to descriptor weights within a training image. A matchconsisting of mainly low-weight descriptors, for example, may have alower adjusted match score than original match score computed withoutassigning weighting factors.

FIG. 7A illustrates an example of common image components 701 a, 701 b.In the example, similar fonts may be used for the headings of twonewspapers and may cause incorrect or irrelevant matches between imagesof the objects. The common image components 701 a, 701 b may appearwithin training images 703 a, 703 b respectively of the two newspapers.The common image components 701 a, 701 b, as illustrated may becharacters of a common word (e.g., the characters “mes” of the word“Times”) appearing within training images 703 a, 703 b.

FIG. 7B illustrates another example of common image components 701 c,701 d. A query image 705 may be matched to two similar training images703 c, 703 d using a matcher or visual object recognition module.Matched regions within the training images 703 c, 703 d may beidentified. However, the matched regions may be identified as commonimage components 701 c, 701 d which may not be specific to the queryimage 705. In one example, the similar training images 703 c, 703 d maybe determined not to be matches to the query image 705. In anotherexample, the ranking of a match score between the query image 705 andthe similar training images 703 c, 703 d may be decreased ordown-weighted based on the identification of common image components 701c, 701 d.

FIG. 8 is a functional block diagram illustrating an example computingdevice 800 used in a computing system that is arranged in accordancewith at least some embodiments described herein. The computing devicemay be a personal computer, mobile device, cellular phone,touch-sensitive wristwatch, tablet computer, video game system, orglobal positioning system, and may be implemented to improve visualobject recognition as described in FIGS. 1-7. In a basic configuration802, computing device 800 may typically include one or more processors810 and system memory 820. A memory bus 830 can be used forcommunicating between the processor 810 and the system memory 820.Depending on the desired configuration, processor 810 can be of any typeincluding but not limited to a microprocessor (μP), a microcontroller(μC), a digital signal processor (DSP), or any combination thereof. Amemory controller 815 can also be used with the processor 810, or insome implementations, the memory controller 815 can be an internal partof the processor 810.

Depending on the desired configuration, the system memory 820 can be ofany type including but not limited to volatile memory (such as RAM),non-volatile memory (such as ROM, flash memory, etc.) or any combinationthereof. System memory 820 may include one or more applications 822, andprogram data 824. Application 822 may include a component matchingalgorithm 823 that is arranged to provide inputs to the electroniccircuits, in accordance with the present disclosure. Program data 824may include content information 825 that could be directed to any numberof types of data. In some example embodiments, application 822 can bearranged to operate with program data 824 on an operating system.

Computing device 800 can have additional features or functionality, andadditional interfaces to facilitate communications between the basicconfiguration 802 and any devices and interfaces. For example, datastorage devices 840 can be provided including removable storage devices842, non-removable storage devices 844, or a combination thereof.Examples of removable storage and non-removable storage devices includemagnetic disk devices such as flexible disk drives and hard-disk drives(HDD), optical disk drives such as compact disk (CD) drives or digitalversatile disk (DVD) drives, solid state drives (SSD), and tape drivesto name a few. Computer storage media can include volatile andnonvolatile, non-transitory, removable and non-removable mediaimplemented in any method or technology for storage of information, suchas computer readable instructions, data structures, program modules, orother data.

System memory 820 and storage devices 840 are examples of computerstorage media. Computer storage media includes, but is not limited to,RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM,digital versatile disks (DVD) or other optical storage, magneticcassettes, magnetic tape, magnetic disk storage or other magneticstorage devices, or any other medium which can be used to store thedesired information and which can be accessed by computing device 800.Any such computer storage media can be part of device 800.

Computing device 800 can also include output interfaces 850 that mayinclude a graphics processing unit 852, which can be configured tocommunicate to various external devices such as display devices 860 orspeakers via one or more A/V ports or a communication interface 880. Thecommunication interface 870 may include a network controller 872, whichcan be arranged to facilitate communications with one or more othercomputing devices 780 over a network communication via one or morecommunication ports 874. The communication connection is one example ofa communication media. Communication media may be embodied by computerreadable instructions, data structures, program modules, or other datain a modulated data signal, such as a carrier wave or other transportmechanism, and includes any information delivery media. A modulated datasignal can be a signal that has one or more of its characteristics setor changed in such a manner as to encode information in the signal. Byway of example, and not limitation, communication media can includewired media such as a wired network or direct-wired connection, andwireless media such as acoustic, radio frequency (RF), infrared (IR) andother wireless media.

Computing device 800 can be implemented as a portion of a small-formfactor portable (or mobile) electronic device such as a cell phone, apersonal data assistant (PDA), a personal media player device, awireless web-watch device, a personal headset device, an applicationspecific device, or a hybrid device that include any of the abovefunctions. Computing device 800 can also be implemented as a personalcomputer including both laptop computer and non-laptop computerconfigurations.

In some embodiments, the disclosed methods may be implemented ascomputer program instructions encoded on a non-transitorycomputer-readable storage media in a machine-readable format, or onother non-transitory media or articles of manufacture. FIG. 9 is aschematic illustrating a conceptual partial view of an example computerprogram product 900 that includes a computer program for executing acomputer process on a computing device, arranged according to at leastsome embodiments presented herein. In one embodiment, the examplecomputer program product 900 is provided using a signal bearing medium901. The signal bearing medium 901 may include one or more programminginstructions 902 that, when executed by one or more processors mayprovide functionality or portions of the functionality described abovewith respect to FIGS. 1-8. Thus, for example, referring to theembodiments shown in FIG. 2, FIG. 4, and FIG. 6 one or more features ofblocks 201-213, blocks 401-405, and blocks 601-607 may be undertaken byone or more instructions associated with the signal bearing medium 901.

In some examples, the signal bearing medium 901 may encompass acomputer-readable medium 903, such as, but not limited to, a hard diskdrive, a Compact Disc (CD), a Digital Video Disk (DVD), a digital tape,memory, etc. In some implementations, the signal bearing medium 901 mayencompass a computer recordable medium 904, such as, but not limited to,memory, read/write (R/W) CDs, R/W DVDs, etc. In some implementations,the signal bearing medium 901 may encompass a communications medium 905,such as, but not limited to, a digital and/or an analog communicationmedium (e.g., a fiber optic cable, a waveguide, a wired communicationslink, a wireless communication link, etc.). Thus, for example, thesignal bearing medium 901 may be conveyed by a wireless form of thecommunications medium 905 (e.g., a wireless communications mediumconforming with the IEEE 902.11 standard or other transmissionprotocol).

The one or more programming instructions 902 may be, for example,computer executable and/or logic implemented instructions. In someexamples, a computing device such as the computing device 800 of FIG. 8may be configured to provide various operations, functions, or actionsin response to the programming instructions 902 conveyed to thecomputing device 800 by one or more of the computer readable medium 903,the computer recordable medium 904, and/or the communications medium905.

It should be understood that arrangements described herein are forpurposes of example only. As such, those skilled in the art willappreciate that other arrangements and other elements (e.g. machines,interfaces, functions, orders, and groupings of functions, etc.) can beused instead, and some elements may be omitted altogether according tothe desired results. Further, many of the elements that are describedare functional entities that may be implemented as discrete ordistributed components or in conjunction with other components, in anysuitable combination and location.

While various aspects and embodiments have been disclosed herein, otheraspects and embodiments will be apparent to those skilled in the art.The various aspects and embodiments disclosed herein are for purposes ofillustration and are not intended to be limiting, with the true scopebeing indicated by the following claims, along with the full scope ofequivalents to which such claims are entitled. It is also to beunderstood that the terminology used herein is for the purpose ofdescribing particular embodiments only, and is not intended to belimiting.

1.-20. (canceled)
 21. A computer-implemented method comprising:receiving a query image; identifying a region of a training image thatcorresponds to the query image; determining, by one or more computers,that the region of the training image corresponds to an annotatedregion, wherein the annotated region is associated with an annotationfor the region of the training image; and in response to determiningthat the region of the training image corresponds to the annotatedregion, providing the annotation.
 22. The method of claim 21, whereinidentifying a region of a training image that corresponds to the queryimage comprises: identifying an object in the query image; andidentifying an object in the region of the training image thatcorresponds to the object in the query image.
 23. The method of claim22, wherein identifying an object in the query image comprises: removingbackground information in the query image to obtain a normalized queryimage; and identifying the object in the normalized query image.
 24. Themethod of claim 21, wherein determining that the region of the trainingimage corresponds to an annotated region comprises determining that theregion of the training image corresponds to the annotated region inresponse to determining that the region of the training image is locatedwithin the annotated region.
 25. The method of claim 21, whereindetermining that the region of the training image corresponds to anannotated region comprises determining that the region of the trainingimage corresponds to the annotated region in response to determiningthat a percentage of the region of the training image is located withinthe annotated region meets a threshold.
 26. The method of claim 21,wherein determining that the region of the training image corresponds toan annotated region comprises determining that a match score between theregion of the training image and the annotated region meets a threshold.27. The method of claim 21, wherein identifying the region of thetraining image that corresponds to the query image comprises identifyinga second region of the training image, wherein the second regioncorresponds to a second annotated region, wherein the second annotatedregion is associated with a second annotation for the second region ofthe training image, and wherein the second annotated region is boundedwithin the annotated region; and wherein providing the annotationcomprises in response to determining that that the query imagecorresponds to the annotated region, providing the annotation and notthe second annotation.
 28. The method of claim 21, wherein theannotation includes metadata associated with an object in the region ofthe training image.
 29. A computer-readable medium storing softwarehaving stored thereon instructions, which, when executed by one or morecomputers, cause the one or more computers to perform operations of:receiving a query image; identifying a region of a training image thatcorresponds to the query image; determining, by one or more computers,that the region of the training image corresponds to an annotatedregion, wherein the annotated region is associated with an annotationfor the region of the training image; and in response to determiningthat the region of the training image corresponds to the annotatedregion, providing the annotation.
 30. The computer-readable medium ofclaim 29, wherein identifying a region of a training image thatcorresponds to the query image comprises: identifying an object in thequery image; and identifying an object in the region of the trainingimage that corresponds to the object in the query image.
 31. Thecomputer-readable medium of claim 30, wherein identifying an object inthe query image comprises: removing background information in the queryimage to obtain a normalized query image; and identifying the object inthe normalized query image.
 32. The computer-readable medium of claim29, wherein determining that the region of the training imagecorresponds to an annotated region comprises determining that the regionof the training image corresponds to the annotated region in response todetermining that the region of the training image is located within theannotated region.
 33. The computer-readable medium of claim 29, whereindetermining that the region of the training image corresponds to anannotated region comprises determining that the region of the trainingimage corresponds to the annotated region in response to determiningthat a percentage of the region of the training image locating withinthe annotated region meets a threshold.
 34. The computer-readable mediumof claim 29, wherein determining that the region of the training imagecorresponds to an annotated region comprises determining that a matchscore between the region of the training image and the annotated regionmeets a threshold.
 35. A system comprising: one or more processors andone or more computer storage media storing instructions that areoperable, when executed by the one or more processors, to cause the oneor more processors to perform operations comprising: receiving a queryimage; identifying a region of a training image that corresponds to thequery image; determining, by one or more computers, that the region ofthe training image corresponds to an annotated region, wherein theannotated region is associated with an annotation for the region of thetraining image; and in response to determining that the region of thetraining image corresponds to the annotated region, providing theannotation.
 36. The system of claim 35, wherein identifying a region ofa training image that corresponds to the query image comprises:identifying an object in the query image; and identifying an object inthe region of the training image that corresponds to the object in thequery image.
 37. The system of claim 36, wherein identifying an objectin the query image comprises: removing background information in thequery image to obtain a normalized query image; and identifying theobject in the normalized query image.
 38. The system of claim 35,wherein determining that the region of the training image corresponds toan annotated region comprises determining that the region of thetraining image corresponds to the annotated region in response todetermining that the region of the training image is located within theannotated region.
 39. The system of claim 35, wherein determining thatthe region of the training image corresponds to an annotated regioncomprises determining that the region of the training image correspondsto the annotated region in response to determining that a percentage ofthe region of the training image locating within the annotated regionmeets a threshold.
 40. The system of claim 35, wherein determining thatthe region of the training image corresponds to an annotated regioncomprises determining that a match score between the region of thetraining image and the annotated region meets a threshold.